[PEP 695] Partial support for new type parameter syntax in Python 3.12 #17233

JukkaL · 2024-05-10T15:58:54Z

Add basic support for most features of PEP 695. It's still not generally useful, but it
should be enough for experimentation and testing. I will continue working on
follow-up PRs after this has been merged.

This is currently behind a feature flag: --enable-incomplete-feature=NewGenericSyntax

These features, among other things, are unimplemented (or at least untested):

Recursive type aliases
Checking for various errors
Inference of variance in complex cases
Dealing with unknown variance consistently
Scoping
Mypy daemon
Compilation using mypyc

The trickiest remaining thing is probably variance inference in cases where some types
aren't ready (i.e. not inferred) when we need variance. I have some ideas about how to
tackle this, but it might need significant work. Currently the idea is to infer variance
on demand when we need it, but we may need to defer if variance can't be calculated,
for example if a type of an attribute is not yet ready. The current approach is to fall
back to covariance in some cases, which is not ideal.

Work on #15238.

JelleZijlstra · 2024-05-10T16:01:37Z

Wonderful, thanks! I'll do a review within the next week.

cdce8p

This is awesome! Thanks for working on it @JukkaL 🚀

Left some comments on stuff I noticed. I've also been able to start testing this change with real code today. Will post a separate comment about it soon.

cdce8p · 2024-05-11T20:19:32Z

mypy/fastparse.py

+if sys.version_info >= (3, 12):
+    ast_TypeAlias = ast3.TypeAlias
+    ParamSpec = ast3.ParamSpec
+    TypeVarTuple = ast3.TypeVarTuple
+else:
+    ast_TypeAlias = Any
+    ParamSpec = Any
+    TypeVarTuple = Any


Maybe use ast_ParamSpec and ast_TypeVarTuple here instead, just to prevent accidental name conflicts in the future?

cdce8p · 2024-05-11T20:23:35Z

mypy/semanal.py

+        for p in s.type_args:
+            upper_bound = p.upper_bound or self.object_type()
+            fullname = self.qualified_name(p.name)
+            type_params.append(
+                (
+                    p.name,
+                    TypeVarExpr(
+                        p.name,
+                        fullname,
+                        [],
+                        upper_bound,
+                        default=AnyType(TypeOfAny.from_omitted_generics),
+                    ),
+                )
+            )
+            all_type_params_names.append(p.name)


Doesn't this need to differentiate between all three TypeVar kinds as well? ParamSpec and TypeVarTuple are supported for type statements.

Checking for p.kind and adding ParamSpecExpr seem to fix Issue 1 + 2.

This is fixed now.

cdce8p · 2024-05-11T20:27:38Z

test-data/unit/check-python312.test

+class C[T]:
+    def m[S](self, x: S) -> T: ...


Wouldn't x be considered Any here as S is only used once? Maybe something like this?

Suggested change

class C[T]:

def m[S](self, x: S) -> T: ...

class C[T]:

def m[S](self, x: S) -> T | S: ...

A good idea, done.

cdce8p · 2024-05-11T20:41:17Z

test-data/unit/check-python312.test

+[case testPEP695ParamSpec]
+# flags: --enable-incomplete-feature=NewGenericSyntax
+from typing import Callable
+
+def g[**P](f: Callable[P, None], *args: P.args, **kwargs: P.kwargs) -> None:
+    f(*args, **kwargs)
+    f(1, *args, **kwargs)  # E: Argument 1 has incompatible type "int"; expected "P.args"


Would be great to test ParamSpec with the type statement here as well.

Added a test case.

cdce8p · 2024-05-11T20:41:34Z

test-data/unit/check-python312.test

+[case testPEP695TypeVarTuple]
+# flags: --enable-incomplete-feature=NewGenericSyntax
+
+def f[*Ts](t: tuple[*Ts]) -> tuple[*Ts]:
+    reveal_type(t)  # N: Revealed type is "Tuple[Unpack[Ts`-1]]"
+    return t


Would be great to test TypeVarTuple with the type statement here as well.

Added a test case.

cdce8p · 2024-05-11T21:07:08Z

As mentioned before, I've started testing this change with Home Assistant. After I saw the PR today, I begun looking through the codebase and converting TypeVar / ParamSpec / TypeVarTuple calls as well as some type aliases to the PEP 695 syntax.

Some general observations
The new syntax is great but not always practical. Turns out it's much more common than I thought that a TypeVar is reused, even imported from another module. This is especially the case when a non-trival upper-bound or constraints are used. Overall I'd say the PEP 695 syntax is useful in about 3/4 of all cases. (Maybe this is an area for future improvements?)

The changes here works fairly well. I only encountered some smaller issues. You mentioned in the description that the work here is incomplete, so you might already plan to address some of them.

Issue 1 - ParamSpec inside type statement

# mypy: enable-incomplete-feature=NewGenericSyntax
from typing import Callable

type _FuncType[**_P] = Callable[_P, None]  # Error

test.py:4: error: The first argument to Callable must be a list of types,
    parameter specification, or "..."  [valid-type]

Edit: This seems to be fixed by checking for PARAM_SPEC_KIND in visit_type_alias_stmt. #17233 (comment)

Issue 2 - Decorator typing with Concatenate and type alias

# mypy: enable-incomplete-feature=NewGenericSyntax
from typing import Callable, Concatenate

class Device: ...

type _FuncType[_T, **_P] = Callable[Concatenate[_T, _P], None]  # Error

def decorator[_DeviceT: Device, **_P](func: _FuncType[_DeviceT, _P]) -> _FuncType[_DeviceT, _P]:
    def wrapped(self: _DeviceT, *args: _P.args, **kwargs: _P.kwargs) -> None:
        ...

    return wrapped

  File "/.../mypy/typeanal.py", line 344, in visit_unbound_type_nonoptional
    assert isinstance(tvar_def, ParamSpecType)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AssertionError:

Edit: This seems to be fixed by checking for PARAM_SPEC_KIND in visit_type_alias_stmt. #17233 (comment)

Issue 3 - maximum semantic analysis iteration count reached

# mypy: enable-incomplete-feature=NewGenericSyntax
from typing import cast

class A:
    def func[T: int](self, value: T) -> T:
        return cast(T, value)

Deferral trace:
    test:6
    ...
    test:6
    test:6
test.py: error: INTERNAL ERROR: maximum semantic analysis iteration count reached

The same issue can be seen here:

# mypy: enable-incomplete-feature=NewGenericSyntax
def func[T](val: T) -> T:
    return unknown()  # Error

Issue 4 - Type alias with forward reference as upper_bound

# mypy: enable-incomplete-feature=NewGenericSyntax
type _FuncType[_T: Device] = _T
class Device: ...

  File "/.../mypy/semanal.py", line 1716, in pop_type_args
    del names[tv.name]
        ~~~~~^^^^^^^^^
KeyError: '_T'

cdce8p · 2024-05-11T21:54:12Z

mypy/semanal.py

+            )
+            all_type_params_names.append(p.name)
+
+        self.push_type_args(s.type_args, s)


Maybe defer if push_type_args returned False e.g. when a forward reference is used as upper_bound? This would fix Issue 4.

Yes, this works. Fixed the issue.

JukkaL · 2024-05-15T15:23:26Z

@cdce8p I think I've fixed all the issues you reported, except for issue 3, which needs support for the new PEP 695 scoping mechanism.

JukkaL · 2024-05-15T16:27:32Z

And I'm thinking of fixing scoping in a follow-up PR.

cdce8p

Thanks @JukkaL 👍🏻 The latest commits indeed fixed all issues. Except for no. 3 but as you mentioned that still needs support for scoping. Makes sense to do that separately. At least with 196b2a3 it doesn't crash anymore.

One area I still get a few errors is with generic classes, overloads and TypeVarTuples. Haven't had time to take a closer look so far, unfortunately.

However, as it's easy enough to work around, just continue using Generic for it, I don't think this should block the PR. If it persists and when I've time to narrow it down, I can always open a new issue.

--
Thanks again for working on this. I look forward to using the new syntax 🚀
Feel free to ping me if I should check out other PRs and test them. Happy to do so.

cdce8p · 2024-05-15T21:35:36Z

Hmm, maybe the overload error is actually related to variance. In the example here _R_co should be inferred as covariant:

from typing import Callable, Generic, TypeVar

class Job[_R_co]:
    def __init__(self, target: Callable[[], _R_co]) -> None:
        self.target = target

def func(
    action: Job[int | None],
    a1: Job[int | None],
    a2: Job[int],
    a3: Job[None],
) -> None:
    action = a1
    action = a2  # error
    action = a3  # error

error: Incompatible types in assignment (expression has type "Job[int]", variable has type "Job[int | None]")  [assignment]
error: Incompatible types in assignment (expression has type "Job[None]", variable has type "Job[int | None]")  [assignment]

It works fine when using Generic and TypeVar:

_S_co = TypeVar("_S_co", covariant=True)

class Job2(Generic[_S_co]):
    def __init__(self, target: Callable[[], _R_co]) -> None:
        self.target = target

def func2(
    action: Job2[int | None],
    a1: Job2[int],
    a2: Job2[int | None],
    a3: Job2[None],
) -> None:
    action = a1
    action = a2
    action = a3

cdce8p · 2024-05-16T05:46:00Z

Hmm, maybe the overload error is actually related to variance. In the example here _R_co should be inferred as covariant:
[...]

After a discussion in a pyright issue, I seems self.target should be marked as Final for it to be covariant. I.e. this works fine with pyright:

from typing import Callable, Generic, TypeVar, Final

class Job[_R_co]:
    def __init__(self, target: Callable[[], _R_co]) -> None:
        self.target: Final = target  # <-- marked as Final

def func(
    action: Job[int | None],
    a1: Job[int | None],
    a2: Job[int],
    a3: Job[None],
) -> None:
    action = a1
    action = a2  # error
    action = a3  # error

Current mypy output:

error: Incompatible types in assignment (expression has type "Job[int]", variable has type "Job[int | None]")  [assignment]
error: Incompatible types in assignment (expression has type "Job[None]", variable has type "Job[int | None]")  [assignment]

JukkaL · 2024-05-16T10:30:05Z

@cdce8p Variance inference should be fixed now. It was broken when using real typeshed stubs. And thanks for the testing and the detailed bug reports -- they've been really helpful!

Also avoid inferring variances repeatedly.

cdce8p · 2024-05-16T12:06:19Z

@cdce8p Variance inference should be fixed now. It was broken when using real typeshed stubs.

Thanks! Can confirm the last commit also fixed the overload issues I saw. AFAICT this PR looks good now 🚀

github-actions · 2024-05-16T12:32:36Z

According to mypy_primer, this change doesn't affect type check results on a corpus of open source code. ✅

JukkaL · 2024-05-16T15:18:54Z

I'll wait one more day before merging, in case somebody else wants to review this.

JelleZijlstra

Thanks, this looks great.

If I'm not mistaken there's some missing serialization code though.

JelleZijlstra · 2024-05-16T15:49:55Z

mypy/fastparse.py

+        for p in type_params:
+            bound = None
+            values: list[Type] = []
+            if isinstance(p, ast_ParamSpec):  # type: ignore[misc]


This can wait for another PR, but we should also check for the .default_value here for PEP 696 support.

We can probably do that in a followup. Should be easy enough to implement support for PEP 696 here as well, although that requires Python 3.13 anyway.

JelleZijlstra · 2024-05-16T15:51:06Z

mypy/nodes.py

    ) -> None:
        super().__init__()
        self.arguments = arguments or []
        self.arg_names = [None if arg.pos_only else arg.variable.name for arg in self.arguments]
        self.arg_kinds: list[ArgKind] = [arg.kind for arg in self.arguments]
        self.max_pos: int = self.arg_kinds.count(ARG_POS) + self.arg_kinds.count(ARG_OPT)
+        self.type_args: list[TypeParam] | None = type_args


This new field should be serialized.

Not sure we actually need that here. Could be missing something but my understanding is that type_args is only used for the SemanticAnalyzer so there isn't a need to serialize it.

If it were necessary, mypy would've probably crashed for me while testing the PR with Home Assistant.

You might be right, I don't actually understand how our serialization works :)

Yeah, this is just part of the syntax and doesn't get used beyond semantic analysis.

cdce8p · 2024-05-17T10:59:40Z

Just released mypy-dev version 1.11.0a2 which is based on 5fb8d62.
It's also published on PyPI so anyone can try it.

https://pypi.org/project/mypy-dev/1.11.0a2/
https://github.com/cdce8p/mypy-dev/releases/tag/1.11.0a2

JukkaL added 30 commits May 3, 2024 11:43

WIP parse type variable definitions in function

725a224

WIP make function type vars work kind of

3058b84

WIP parse new-style type params in classes

fc23bcf

WIP make class type vars work partially

61e971d

WIP variance test case

47cb82d

WIP first attempt at inference of variance

e56dd96

WIP infer variance on demand

21c3eb6

Fall back to covariance if variance not ready, and support attributes

7e15173

Fix join

b76e7eb

Test meet when not ready

650468b

WIP minimal support for inheritance

639a85b

Add inheritance test cases

2b4dbe1

Add test cases

5081cb9

Support protocols

d42026e

Add generic function test cases

9745e5b

Basic support for PEP 695 type aliases

ddb4f58

Partial support for upper bounds

a1ea487

Ssupport forward refs in upper bounds

0e41f88

Fix for invalid upper bound in generic functions

16ffbfb

Add test case

c6bbe70

Add missing visit methods

59434fa

Generate error in mypyc if compiling a type alias statement

84e3f11

Fix type alias self check

586eb64

Fix variance not ready handling in typestate

6bbc046

Fix missing return value in infer_class_variances

ea39e96

Fix false positives about undefined names

ad8c062

Basic support for tuple subclasses

6138266

Black

98bd612

Refactor AST

a172027

Support value restriction

c46550a

Minor tweaks

d8291f3